Augmented reality systems provide information relevant to a person's current environment by adding rendered images of synthetic objects (e.g., three-dimensional (3D) assets) to a scene of an environment currently being viewed by the person. In one example, a designer or prospective purchaser can add furniture to an image or video of a room and adjust the position of the furniture in the room. The furniture can be added to a video as the designer or prospective purchaser walks around the room, to view different perspectives of the furniture in the room. Hence, the designer or prospective purchaser can select appropriate furniture prior to purchase, without physically moving the furniture in the room.
The augmented reality scene can be experienced through an electronic device, such as a smart phone, where the user is able to see the augmented reality scene when looking at the display of the smart phone while holding the smart phone in the environment. A camera on the smart phone is used for video see-through. Alternatively, the augmented reality scene can be experienced through an optical device, such as a head-mounted display with a lens (e.g., goggles), on which the user sees the augmented reality scene.
To be of practical value in many applications, such as when influencing a design or purchasing decision, the inserted synthetic object must be combined with the real-world environment so that the composite image of the augmented reality scene is photorealistic (e.g., the inserted synthetic object cannot be distinguished from natural objects in the real-world environment). Hence, the inserted synthetic object must match the environment in which it is inserted by accounting for lighting in the environment and occlusion with the background where the object is inserted. Additionally, the environment may need to be adjusted based on the inserted object (e.g., due to shadows cast by the inserted object). Furthermore, processing should be done in real time with no appreciable delay to the user.
Traditional methods of creating rendered images of synthetic objects to be inserted into an augmented reality experience guess at, and therefore do not accurately represent, the lighting of the environment. Consequently, traditional methods are unable to accurately match the lighting of the inserted synthetic object, such as with shading and lighting of the object, to the environment the objected is inserted in. Moreover, traditional methods do not accurately represent the combination of object material properties and the lighting of the environment in real time. Without an accurate real-time representation of lighting in the environment, traditional methods are unable to accurately adjust the environment due to the inserted object, such as by accounting for dynamically cast shadows of the inserted object in a video containing multiple perspectives of the inserted object. Moreover, traditional methods do not support occlusion with the background where the synthetic object is inserted. As a result, traditional methods of creating rendered images of synthetic objects and inserting them into an augmented reality scene (e.g., picture or video) do not meet the real-time requirements of an augmented reality system needed for many applications, such as influencing design and purchasing decisions.